Abstract
Background: Decision-making under deep uncertainty requires robust methodological approaches that can handle incomplete information and multiple stakeholder perspectives. This review systematically maps the methodological landscape across 15,000+ publications.
Methods: We employ automated text mining, keyword classification, and bibliometric analysis to categorize methods, applications, and trends in decision support under uncertainty.
Keywords: decision support, deep uncertainty, value of information, Bayesian analysis, stochastic optimization, multi-criteria decision analysis, systematic review, bibliometrics
Decisions made under uncertainty occur in circumstances where a decision-maker is in the position of choosing between potentially competing options. Such decisions often have to be made with no formal scientific support and often do not need much formal support (Keeney, 2004). Decision-supporting research can offer formal support by prescribing a recommended course of action. In general, this is done with the assumption that decision-makers seek maximum expected utility. This means we must assume that the decision maker will want the option with the greatest positive outcome according to their stated outcomes of interest. The decision-making processes can be categorized into two levels of decision making: 1) a decision aimed at influencing an outcome where the expected choice is the one which maximizes the desired outcome, based on information currently available to the decision maker and 2) a decision on how to allocate resources to reduce the uncertainty in the previous decision problem, i.e., to increase the current information to improve the likelihood of choosing the option with the most beneficial outcome.
Such decisions are often made in policy settings like in healthcare, where a choice to fund the health of one part of society often comes at the cost of another, assuming limited budget allocation (Claxton et al., 2015). This choice involves costs, benefits and risks in the form of possible damage and trade-offs in comparison to a baseline and or other decision options.
Our own work has covered such situations regarding the nutritional costs of agricultural expansion in Uganda (C. W. Whitney et al., 2018), the choice between different siltation management options in Burkina Faso (Lanzanova et al., 2019), the management of competing water resources in Kenya (Luedeling et al., 2015), and deciding between agroforestry interventions in Vietnam (Do et al., 2020). In these cases, we apply, adopt and update the classical processes of Decision Analysis (Howard & Abbas, 2015; Jeffrey, 1990; Keeney, 1982). To achieve this, we co-generate impact pathways of decisions to provide the structure for a mathematical formula that is then used to estimate the expected utility. We perform a quantitative analysis (Luedeling et al., 2022) of welfare-based decision-making processes using Monte Carlo simulations. In our work, we interpret the welfare-based decision problem as a von Neumann-Morgenstern utility function and apply the ‘Value of Information Analysis’ to assign a value to a certain reduction in uncertainty or, equivalently, increase in information.
There are a number of ways that decisions made under uncertainty can be supported. Through this review we seek to generate a complete overview of all the applications of similar processes that aim to support decision-making under uncertainty.
We offer an overview of methods that are applied to support decisions under uncertainty. We focus on summarizing the state-of-the-art methods in decisions. We identify, define, and evaluate research by extracting the decision-supporting methods, describing them and their approach to supporting decisions. We present a protocol-driven comprehensive review and synthesis of the various methods applied for decision support. We offer an overview of where these methods are currently applied and some interpretation about where these applications can be expanded.
Protocol: We follow the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) (Figure 2). We gather data from the Web of Science Core Collection curated database of published, peer-reviewed and larger and more diverse published articles, preprints, theses, books, and other relevant scientific content in Google Scholar. After much deliberation and discussion amongst the authors, the final search query consisted of the following keywords.
Decision + Intervention OR policy + Uncertainty + Expert OR stakeholder + Model OR monte carlo OR simulation OR computer assisted + value of information OR information accuracy
“decision”+(“intervention”OR”policy”)+“uncertainty”+(“expert”OR”stakeholder”)+(“model”OR”monte carlo”OR”simulation”OR”Bayesian”OR”computer assisted”)+ (“value of information”OR”information accuracy”)
From Web of Science, 14 records were collected in May 2023, and from Google Scholar, 17500+ records were collected in April and May 2023. Associated papers and articles from the search records were also gathered as secondary data collection. Some had shared arXiv or doi links, other parts of shared contributions and assorted affiliations that led to new papers. We applied specific inclusion criteria to this large set of records to reduce the number of records covered under this study. An independent screening of the collected records is performed to confirm their eligibility for the study.
We use Keeney’s representation of where formal decision support is required and applied as motivation and framework for the assessment - Keeney’s judgment about how 10,000 decisions are typically made - his personal histogram of 10,000 decisions being faced by numerous decision makers @ref(#fig:01_keeney) (Keeney, 2004).
We followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) @ref(fig:02_prisma) SystematicReviewOverviewFigure
We preserved our search string on searchRxiv (C. Whitney, 2023)
Google Scholar (GS) yielded 17,600 results but without an asterisk for decision* and model* it yielded just 17,400 results. This is a bit counter intuitive; the ’*’ should widen rather than narrow the search. If anything, we should expect the opposite effect and the larger return in the former search. Then these 200 additional results would say something like ‘models’ or ‘decisions’ but not ‘model’ or ‘decision’. These should be kept.
‘Google Scholar advanced search’ is much less targeted (2.4 million hits): with all of the words: decision intervention policy uncertainty model with at least one of the words: “value of information”, “information accuracy”, “intervention”, “policy”, “model*“,”monte carlo” “simulation”, “Bayesian”, “computer assisted”, “expert”, “stakeholder”, where my words occur ‘anywhere in the article’.
After importing papers to Zotero we:
In looking for the PDFs and following up with some of the thesis, notes, syllabi etc. in the Google Scholar returned list we added additional associated papers and articles. Some had shared arXiv or doi links, other parts of shared contributions and assorted affiliations that lead to new papers. Sometimes the class notes and syllabus that were in the scholar search return were there because they listed papers that were relevant to our search terms. These were sometimes already in our main collection and sometimes not. When they were not in our collection, we added them.
Web of Science - 14 results
CABI - 13 results
Decision + Intervention OR policy + Uncertainty + Expert OR stakeholder + Model OR “monte carlo” OR simulation OR “computer assisted” + “value of information” OR “information accuracy”
We import bibliographic records using the bib2df package (Ottolinger, 2024), converting our BibTeX file into a structured data frame for systematic processing.
# Read the bib file
bib_data <- bib2df::bib2df("bib/23_Methods_Review_Holistic_Systems.bib")
Our BibTeX contains (n = 15200) citations.
We prepare the data using dplyr (Wickham et al., 2023), for annotation tracking and text consolidation.
# Basic cleaning and text preparation
clean_bib_data <- bib_data %>%
dplyr::mutate(
has_annotation = !is.na(ANNOTE) & ANNOTE != "",
annotation_text = ifelse(has_annotation, ANNOTE, ""),
full_text = paste(TITLE, ANNOTE, sep = " "),
year = as.numeric(YEAR)
)
(3499 of the citations have annotations.
We search the bibliographic record for methodological terms from titles, abstracts, keywords, and annotations.
library(tidytext)
# Combine all text fields for comprehensive analysis
all_text_data <- clean_bib_data %>%
mutate(
combined_text = paste(
TITLE,
ifelse(!is.na(ABSTRACT), ABSTRACT, ""),
ifelse(!is.na(KEYWORDS), KEYWORDS, ""),
ifelse(!is.na(ANNOTE), ANNOTE, ""),
sep = " "
)
)
# Tokenize all text fields to discover method terms
method_terms_comprehensive <- all_text_data %>%
select(combined_text) %>%
unnest_tokens(word, combined_text) %>%
count(word, sort = TRUE) %>%
filter(n > 5, nchar(word) > 3) %>% # Filter for meaningful terms
anti_join(stop_words) # Remove common stop words
| Term | Frequency |
|---|---|
| information | 6344 |
| decision | 3187 |
| management | 2912 |
| analysis | 2572 |
| research | 2373 |
| data | 2186 |
| study | 2166 |
| model | 2159 |
| based | 2123 |
| risk | 2033 |
| supply | 1871 |
| date | 1778 |
| export | 1755 |
| cited | 1742 |
| 2024 | 1731 |
| publisher | 1638 |
| health | 1551 |
| chain | 1525 |
| systems | 1483 |
| cost | 1456 |
| approach | 1431 |
| uncertainty | 1429 |
| performance | 1281 |
| system | 1227 |
| bayesian | 1191 |
| economic | 1184 |
| technology | 1153 |
| knowledge | 1113 |
| review | 1054 |
| social | 1037 |
# Analyze multi-word method terms using n-grams
method_bigrams_comprehensive <- all_text_data %>%
select(combined_text) %>%
unnest_tokens(bigram, combined_text, token = "ngrams", n = 2) %>%
count(bigram, sort = TRUE) %>%
filter(n > 2) # Keep only terms appearing multiple times
| Bigram | Frequency |
|---|---|
| of the | 4200 |
| in the | 3119 |
| cited by | 1733 |
| 05 may | 1730 |
| 2024 cited | 1730 |
| date 05 | 1730 |
| export date | 1730 |
| may 2024 | 1730 |
| on the | 1586 |
| and the | 1426 |
| supply chain | 1419 |
| of information | 1399 |
| decision making | 1294 |
| to the | 1225 |
| value of | 1202 |
| for the | 1123 |
| of a | 897 |
| based on | 779 |
| in a | 755 |
| this paper | 727 |
# Use pattern matching to find potential method terms in context
method_patterns <- c(
"analysis", "model", "method", "approach", "framework",
"technique", "algorithm", "estimation", "evaluation", "simulation",
"optimization", "bayesian", "statistical", "stochastic"
)
# Extract sentences containing methodological terms
method_sentences <- all_text_data %>%
select(combined_text) %>%
unnest_tokens(sentence, combined_text, token = "sentences") %>%
filter(str_detect(tolower(sentence), paste(method_patterns, collapse = "|"))) %>%
sample_n(10) # Sample for review
| Example_Sentences |
|---|
| the model predicts that individuals acquire more information when the stakes are higher. |
| an evaluation of the measurement properties within an analysis of covariance structures framework indicated that the operational measures developed here largely satisfy the criteria for unidimensionality, convergent, discriminant, and predictive validity. |
| conventional methods like grid search and random search can be computationally demanding. |
| development of a new cost-effectiveness model this chapter provides a detailed description of the modelling approach, the estimation of input parameters used to populate the model and the key assumptions underpinning the cost-effectiveness results. |
| multi-attribute identifiability analysis also yielded optimal parameter values that were temporally less variable than those estimated using streamflow alone. |
| the role of value for money in public insurance coverage decisions for drugs in {australia}: a retrospective analysis 1994-2004 |
| we show that this model can be systematically approximated by a single-location inventory problem. |
| the development of the {economic} impacts of {smoking} {in} {pregnancy} ({esip}) model for measuring the impacts of smoking and smoking cessation during pregnancy |
| embedding security messages in existing processes: {a} pragmatic and effective approach to information security culture change. |
| the incorporation of mechanism improves the confidence in predictions made for a variety of conditions, while the statistical methods provide an empirical basis for parameter estimation and allow for estimates of predictive uncertainty. |
# Specifically analyze author-provided keywords
if("KEYWORDS" %in% names(clean_bib_data)) {
keyword_analysis <- clean_bib_data %>%
filter(!is.na(KEYWORDS)) %>%
select(KEYWORDS) %>%
unnest_tokens(keyword, KEYWORDS, token = "regex", pattern = ";|,") %>%
mutate(keyword = str_trim(tolower(keyword))) %>%
count(keyword, sort = TRUE) %>%
filter(n > 1, nchar(keyword) > 2)
}
| Keyword | Frequency |
|---|---|
| bayesian | 550 |
| read cw | 195 |
| monte carlo simulation | 155 |
| bin | 139 |
| bayesian decision analysis | 66 |
| stanford | 46 |
| supply chain management | 36 |
| value of information | 33 |
| machine learning | 17 |
| information technology | 14 |
| risk management | 14 |
| supply chain | 14 |
| evpi | 13 |
| resilience | 13 |
| social science / sociology / general | 11 |
| trust | 11 |
| risk assessment | 10 |
| social media | 10 |
| optimization | 9 |
| review | 9 |
# method categories
method_categories <- list(
bayesian = c("bayesian", "bayes", "mcmc", "markov chain", "prior", "posterior"),
simulation = c("simulation", "monte carlo", "stochastic", "agent-based", "discrete event"),
optimization = c("optimization", "linear programming", "nonlinear programming", "heuristic"),
statistical = c("regression", "anova", "time series", "survival analysis", "mixed model"),
decision_analysis = c("decision analysis", "decision tree", "markov model", "value of information", "voi"),
machine_learning = c("machine learning", "neural network", "random forest", "svm", "clustering"),
multi_criteria = c("multi-criteria", "multi criteria", "analytic hierarchy", "ahp", "topsis"),
economic_evaluation = c("cost-effectiveness", "cost-benefit", "cost-utility", "economic evaluation"),
risk_analysis = c("risk analysis", "risk assessment", "sensitivity analysis", "uncertainty analysis")
)
We apply automated classification using a custom dictionary-based algorithm (detect_methods_enhanced.R) that identifies methodological approaches from text content. The purrr package (Wickham & Henry, 2025) enables efficient vectorized processing across all records. Our dual detection (full text and titles only) enables confidence scoring, where methods appearing in titles receive higher reliability weights for subsequent analysis.
source("R/detect_methods_enhanced.R")
# Apply method detection
clean_bib_data <- clean_bib_data %>%
mutate(
detected_methods = map_chr(full_text, detect_methods_enhanced),
methods_from_title = map_chr(TITLE, detect_methods_enhanced)
)
We assign confidence levels to method classifications based on detection location. Methods identified in titles receive higher confidence than those found only in annotations, enabling quality-weighted analysis.
source("R/add_confidence_scores.R")
# Apply confidence scoring
clean_bib_data <- add_confidence_scores(clean_bib_data)
We validate the automated classification by reporting key metrics including detection rates and confidence level distribution.
| Metric | Count |
|---|---|
| Total papers | 15200 |
| Papers with methods detected | 1828 |
| Papers with VOI | 263 |
| High confidence classifications | 1823 |
| Medium confidence classifications | 5 |
| Low confidence classifications | 0 |
We display representative examples of automated method classifications to illustrate detection accuracy and confidence assignment.
# Show some examples
sample_results <- clean_bib_data %>%
filter(detected_methods != "") %>%
select(TITLE, detected_methods, method_confidence) %>%
head(10)
print(sample_results)
## # A tibble: 10 × 3
## TITLE detected_methods method_confidence
## <chr> <chr> <chr>
## 1 A hierarchical bayesian approach for inco… bayesian high
## 2 Integrating monitoring and optimization m… optimization high
## 3 A systematic review and economic evaluati… economic_evalua… high
## 4 The rocky road to extended simulation fra… simulation; opt… high
## 5 A practical guide to value of information… decision_analys… high
## 6 Late pregnancy ultrasound to screen for a… decision_analys… high
## 7 Efficient research design: {Using} value-… economic_evalua… high
## 8 Economic evaluation of transperineal vers… economic_evalua… high
## 9 The cost-effectiveness of a novel {SIAsco… economic_evalua… high
## 10 Bayesian design and analysis of external … bayesian high
We quantify and visualize the distribution of methodological approaches across the literature using frequency analysis and bar chart visualization via ggplot2 (Wickham et al., 2025).
source("R/analyze_method_frequency.R")
method_results <- analyze_method_frequency(clean_bib_data)
print(method_results$plot)
(#fig:method_results_plot)TRUE
print(method_results$counts)
## # A tibble: 9 × 2
## detected_methods n
## <chr> <int>
## 1 economic_evaluation 433
## 2 decision_analysis 396
## 3 bayesian 380
## 4 simulation 238
## 5 risk_analysis 209
## 6 optimization 169
## 7 multi_criteria 77
## 8 machine_learning 64
## 9 statistical 22
The method_results reveal dominant methodological paradigms and identifies less common approaches.
We conduct specialized analysis of Value of Information (VOI) literature, examining methodological patterns and temporal trends.
source("R/analyze_voi_papers.R")
# Analyze VOI papers
voi_results <- analyze_voi_papers(clean_bib_data)
## === VOI PAPERS ANALYSIS ===
## Number of VOI papers: 263
##
## Methods used in VOI papers:
## # A tibble: 6 × 2
## detected_methods n
## <chr> <int>
## 1 decision_analysis 263
## 2 economic_evaluation 20
## 3 bayesian 5
## 4 simulation 4
## 5 optimization 2
## 6 risk_analysis 2
(#fig:voi_results)TRUE
analyze_voi_papers shows the methodological evolution and application contexts specific to VOI.
We compile all classification results into a structured results_table and export for external validation and secondary analysis.
# Create a comprehensive results table
results_table <- clean_bib_data %>%
select(
TITLE,
YEAR = year,
HAS_ANNOTATION = has_annotation,
DETECTED_METHODS = detected_methods,
CONFIDENCE = method_confidence,
VOI = has_voi,
BAYESIAN = has_bayesian,
SIMULATION = has_simulation,
UNCERTAINTY = has_uncertainty
)
# Export for manual verification
write.csv(results_table, "data/method_detection_results.csv", row.names = FALSE)
We provide a comprehensive summary of the automated classification pipeline performance and output metrics. The summary quantifies analysis scope and classification reliability.
# Create summary report
summary_stats <- clean_bib_data %>%
summarise(
total_papers = n(),
papers_with_methods = sum(detected_methods != ""),
high_confidence = sum(method_confidence == "high")
)
knitr::kable(summary_stats, caption = "Analysis Summary")
| total_papers | papers_with_methods | high_confidence |
|---|---|---|
| 15200 | 1828 | 1823 |
We identify the most prevalent methodological approaches using frequency analysis and tidyr (Wickham et al., 2024) for data restructuring.
# Show top methods
top_methods <- clean_bib_data %>%
filter(detected_methods != "") %>%
separate_rows(detected_methods, sep = "; ") %>%
count(detected_methods, sort = TRUE) %>%
head(10)
print(top_methods)
## # A tibble: 9 × 2
## detected_methods n
## <chr> <int>
## 1 economic_evaluation 433
## 2 decision_analysis 396
## 3 bayesian 380
## 4 simulation 238
## 5 risk_analysis 209
## 6 optimization 169
## 7 multi_criteria 77
## 8 machine_learning 64
## 9 statistical 22
top_methods reveals the methodological ecosystem’s core components, highlighting established practices and potential gaps in decision support under uncertainty literature.
We examine Value of Information (VOI) papers specifically to assess classification plausibility and confidence distribution within this focused methodological domain.
# Quick validation - check if our detection makes sense
validation_check <- clean_bib_data %>%
filter(has_voi) %>%
select(TITLE, detected_methods, method_confidence) %>%
arrange(desc(method_confidence))
print(validation_check)
## # A tibble: 263 × 3
## TITLE detected_methods method_confidence
## <chr> <chr> <chr>
## 1 A practical guide to value of information… decision_analys… high
## 2 Late pregnancy ultrasound to screen for a… decision_analys… high
## 3 A cost-effectiveness and value of informa… decision_analys… high
## 4 Expected value of information analysis to… decision_analys… high
## 5 Are head-to-head trials of biologics need… decision_analys… high
## 6 Efficient value of information computation decision_analys… high
## 7 Supplier quality improvement: {The} value… decision_analys… high
## 8 Use of value of information in {UK} healt… decision_analys… high
## 9 Decisions on further research for predict… decision_analys… high
## 10 Towards social learning in water related … decision_analys… high
## # ℹ 253 more rows
validation_check ensures classification accuracy.
We examine the subset of classifications receiving the highest confidence rating, where methods were detected in both titles and full text.
# Check papers with high confidence
high_confidence <- clean_bib_data %>%
filter(method_confidence == "high") %>%
select(TITLE, detected_methods, methods_from_title)
print(high_confidence)
## # A tibble: 1,823 × 3
## TITLE detected_methods methods_from_title
## <chr> <chr> <chr>
## 1 A hierarchical bayesian approach for inc… bayesian bayesian
## 2 Integrating monitoring and optimization … optimization optimization
## 3 A systematic review and economic evaluat… economic_evalua… economic_evaluati…
## 4 The rocky road to extended simulation fr… simulation; opt… simulation; optim…
## 5 A practical guide to value of informatio… decision_analys… decision_analysis
## 6 Late pregnancy ultrasound to screen for … decision_analys… decision_analysis…
## 7 Efficient research design: {Using} value… economic_evalua… economic_evaluati…
## 8 Economic evaluation of transperineal ver… economic_evalua… economic_evaluati…
## 9 The cost-effectiveness of a novel {SIAsc… economic_evalua… economic_evaluati…
## 10 Bayesian design and analysis of external… bayesian bayesian
## # ℹ 1,813 more rows
We provide an interactive reference browser using the DT package (Xie et al., 2025) for easy navigation of the analyzed literature collection.
# Minimal interactive table
DT::datatable(
clean_bib_data %>%
select(
Author = AUTHOR,
Year = YEAR,
Title = TITLE
) %>%
arrange(desc(Year), Author),
options = list(
pageLength = 5,
lengthMenu = c(5, 10, 20),
dom = 'tip' # table, info, pagination only
),
caption = "Reference List",
rownames = FALSE,
filter = "none" # remove filter to save space
)
(#fig:datatable_clean_bib_data)TRUE